% arara: pdflatex: {files: [MathSACpr2014]}
% !arara: indent: {overwrite: yes}
\chapter[Outcomes and Assessment]{Reflect on learning outcomes and assessment,
  teaching methodologies, and content in order to improve the quality of teaching,
  learning and student success.}\label{chap:outcomes}
\epigraph{Learning outcomes assessment is key to addressing both affordability and access issues}
{Knowing What Students Know and Can Do \cite{outcomesquote}}

\section[Course-level outcomes]{Identify and give examples of assessment-driven
  changes made to improve attainment of course-level student learning outcomes.
  Where key sequences exist, also include information about assessment-driven changes to those sequences.} 

The SAC has mostly concentrated on college level outcomes in the last five years for the following two reasons:
\begin{enumerate}
	\item The Curriculum Committee currently requires an ``out there'' (\cite{courseoutcomes}) focus for course-level learning outcomes, with no requirement that outcomes be assessable or measurable.
	\item The annual assessment reporting for the Learning Assessment Council (LAC) has focused on the College's Core Outcomes, not course outcomes.
\end{enumerate}

However, we are fortunate to have math faculty involved with the Learning Assessment Council (LAC) and the Curriculum Committee.
This involvement has kept our SAC aware of the college's ongoing discussions regarding a possible future change for the focus of course-level learning outcomes  (i.e., expectation of measurability) and related accreditation standards (e.g., \cite[Standard 4.A.3]{NWCCU}).

Many of our current learning outcomes were developed to satisfy the requirements of an ``out there'' focus and this has resulted in oddly worded or aspirational outcomes.
Here are two examples from MTH 251 (Calculus 1): 

\begin{quote}
	Appreciate derivatives and limit-related concepts that are encountered in the real world, [and] understand and be able to communicate the underlying mathematics involved to help another person gain insight into the situation.
\end{quote}
\begin{quote}
	Enjoy a life enriched by exposure to Calculus.
\end{quote}

While we hope that our students will be able to ``help another person gain insight into the situation'' and that they will ``enjoy a life enriched by exposure to Calculus,'' we recognize that outcomes like these are not easily measured.
Before our SAC can make assessment-driven changes to improve students' attainment of outcomes, we need to first develop measurable outcomes that represent the intent of our courses.

In 2012/13 the MTH 60/65 CCOG subcommittee decided to develop course-level outcomes that were meaningful and assessable; this was done despite concerns that Curriculum Committee might reject them because they lacked the required ``out there'' focus.
The Math SAC was largely supportive of the shift to assessable outcomes, suggesting that we are supportive of the college transitioning toward a culture assessing outcomes at all levels.

While crafting the new (\emph{draft}) course outcomes, the MTH 60/65 CCOG subcommittee connected each proposed outcome to an actual assessment activity used by a member of the committee in order to ensure that each outcome was truly measurable.
What follows are a few of the resultant (\emph{draft}) outcomes.
\begin{description}
	\item[Argument Construction:] Construct and judge the validity of an argument.
		(e.g., Why does a particular symbolic representation match a particular
		graphical representation?)

	\item[Representational Fluency:] Demonstrate the ability to distinguish
		different meanings of `variable', e.g., a variable can represent a varying
		quantity in an expression or an unknown quantity in an equation.

	\item[Problem Solving:] Use appropriate (mathematical) tools in the context of
		problem solving, modeling, interpreting, etc.
		(Know what approach to take, what information you have, what information you need, what techniques you have to solve the problem, what the graph tells you, what the formula tells you, what model you can build).
\end{description}

The current layout of the CCOGs does not differentiate critical content from less critical content.
To address this, the committee discussed the option of formatting the content area of the CCOGs in a pyramid structure with the most critical content highlighted at the bottom of the pyramid.

The current CCOGs do not align the content to the course outcomes.
To help make the CCOG a better communication tool, the committee discussed making explicit connections between the course content and the course outcomes as well as explicit connections between the course outcomes and the college's core outcomes.
This work was in progress when we realized that this shift in outcomes should not be done in isolation from other courses in the pre-college math sequence; in response, we formed a DE Math Subcommittee with the goal of creating vertically aligned outcomes that created a coherent progression from MTH 20 through MTH 95.
Since its formation in June 2013 there has been a major shift in how we are hoping to approach DE level math courses.
As the Math SAC considers new math pathways for DE level students, the CCOGs for these new courses will likewise be explicitly aligned and measurable.

In summary, we are interested in developing rich, meaningful, and measurable outcomes that better represent the intent of our courses than our current outcomes do.
Since course outcomes are required in syllabi, properly representing the intended focus for the course in the course outcomes is critical.
We hope that the conversations in the LAC and Curriculum Committee concerning course-level learning outcomes will lead to Curriculum accepting more flexible wording of course outcomes in the near future.

\recommendation[Learning Assessment Council, Curriculum Committee]{The college should adopt a new model
	for course-level learning outcomes that allow for outcomes that are more
	descriptive of the actual content in the course and that are also better suited
	for student-level and course-level assessment.}

\section{Addressing College Core Outcomes}

\subsection[Outcomes in courses and the math program]{Describe how each of the
	College Core Outcomes are addressed in courses, and/or aligned with program
	and/or course outcomes.}

The college's Core Outcomes may be found at \cite{coreoutcomes}.

\begin{description}
	\item[Communication] is stated in many of our CCOGs as a course outcome.
		We believe it is important for students to communicate ideas using mathematics in a meaningful manner through appropriate use of notation and concise, accurate statements.
		Here are some excerpts from course outcomes in MTH CCOGs:
		\begin{aquote}{MTH 111--112, MTH 211--213, MTH 251--254} {\ldots}and then interpret and clearly communicate the results.
		\end{aquote}

		\begin{aquote}{MTH 243--244}
			{\ldots}and clearly interpret the results via written or oral communication.
		\end{aquote}

		\begin{aquote}{MTH 111--112, MTH 243--244, MTH 251--254}
			{\ldots}understand and be able to communicate the underlying mathematics
			involved to help another person gain insight into the situation.
		\end{aquote}

	\item[Community and Environmental Responsibility] is not directly addressed in
		our courses or outcomes.
		Our Social Justice Workgroup (see \cpageref{cur:sub:socialJustic} and \vref{app:sec:socialJustic}) discusses issues related to teaching with community and environmental responsibility in mind.
		Also, sections of MTH 111H (see \cpageref{cur:sub:111H}) have a related component involving tutoring math to someone in a student's community.
		Examples of Service Learning components used by some faculty are given on \cpageref{other:sec:servicelearning}.

	\item[Critical Thinking and Problem Solving] is fundamental to mathematics so, at first glance, it may seem that there would
		be easy agreement between mathematics faculty for how this Core Outcome is
		present in our curriculum.
		This is not the case.

		Some PCC math faculty believe that everything in a math course is a manifestation of this Core Outcome, including practice problems intended to develop procedural fluency in a non-contextual setting (e.g., solve the quadratic equation by the Zero Product Property).
		Indeed, one can certainly witness the critical analysis required within the realm of procedural fluency when students struggle with  mathematical abstraction.
		This happens at the most basic level, especially for students who are not detail-oriented or unable to easily generalize abstractly in a non-contextual manner.
		It is also present at higher levels of math when students are faced with skill-based problems that require integration of several previously learned procedures.

		Traditionally, the main focus of most undergraduate math programs is developing a students' algebraic procedural fluency (including our current curriculum).
		While procedural fluency is important, some PCC math faculty wish to narrow the focus of this Core Outcome in a way that goes beyond procedural fluency.
		Some examples include the development of students' ability to solve meaningful contextualized problems, explain concepts and justify results in a mathematically sound manner, analyze mathematical patterns, or create elegant proofs.

		Assessment work has made us aware of how differently we can view critical thinking and problem solving, and it has developed our thinking considerably.
		This will remain an ongoing conversation.
		At this time we have chosen to take the broad view of Critical Thinking and Problem Solving that includes procedural fluency.

	\item[Cultural Awareness] is not formally covered in any of our courses.
		Some individual instructors may incorporate elements of cultural awareness into their classes.
		Some examples of how cultural awareness could be addressed by individual math instructors instructors are:
		\begin{itemize}
			\item differences in notations and procedures used by different cultures;
			\item similarities and differences with regard to the use of mathematics as a tool;
			\item attitudes toward mathematics in various cultures (e.g., math anxiety is not a worldwide issue);
			\item history of mathematical  development in various cultures and how it was influenced by dominant philosophical attitudes of the region;
			\item use of mathematics as an agent of social change by shining a light on inequity with regard to treatment of, and resource allocation for, marginalized populations as compared to the dominant culture; the social justice workgroup considers this---see \cpageref{cur:sub:socialJustic} and \vref{app:sec:socialJustic}.
			      %  I was reading recently (in the book ``Lean In'' which as it happens Chris%  Chairsell gave me to read, though that book references other studies if you%  need the original research) that women are less likely than men to raise%  their hands in class, though they often know the answer as well as or better%  than the men in class.  Apparently this is also true of minority students.%  You may want to consider as a SAC creating a new paradigm in future classes%  banning hand-raising as a method of encouraging class participation, because%  if anyone self-suppresses then the practice is ineffective to its very%  purpose (of encouraging participation).  Bottom line:  there is more to%  cultural awareness than teaching students the history of math.
		\end{itemize}

	\item[Professional Competence] is described by PCC as the ability to
		``demonstrate and apply the knowledge, skills and attitudes necessary to enter
		and succeed in a defined profession or advanced academic program.'' As a
		discipline that builds upon prior knowledge and skills, successful completion
		of any of our sequenced courses prepares students for the next math course.
		A certain level of mathematical competency is valued by PCC and is required for all of our degree-seeking students.
		Currently, most students show math competency by taking a math course at PCC.
		In this way, the Math SAC plays a critical role in the professional competence for students' educational goals whether it be for a defined profession or advanced academic program.
		Discussions about the necessary mathematical knowledge, skills and attitudes for the wide range of educational goals of PCC's students are ongoing.
		% Consider adding a whole new paragraph that maybe starts ``Additionally, we
		% hope to offer new new DE level courses that will focus on STEM applications
		% as an integral part of learning math.  This would improve student ...

		Additionally, we offer one sequence that directly addresses professional competence within math education:  Our  MTH 211--213 ``Foundations of Elementary Mathematics'' sequence of courses are taken by students interested in pursuing a career in teaching in the K--12 education system.
		At minimum, two of the three courses are prerequisites for obtaining a degree in Education in Oregon.
		Each course emphasizes specific topics of mathematical theory that are the basic building blocks of mathematics instruction in the K--12 system.
		Some instructors require students to maintain a portfolio, learn multiple assessment techniques, do field observations of teachers in the K--12 system, learn about the Common Core Standards and trends in education both state and nation-wide, and are given guidance in areas such as preparing for the CBEST and PRAXIS as well as decision making in the various avenues for pursuing a degree in education.
		This exposure to what the field of education actually entails helps students make sound career choices early on in their academic pursuits.
	\item[Self-Reflection] is an outcome that we believe students develop in the course of their
		educational experience.
		Many Math SAC faculty believe we can assist this development; others question if this should be a responsibility of faculty (or if this should even be a Core Outcome of the college).
		While the SAC needs more discussion to develop a shared understanding of what Self Reflection means in the context of a mathematics course and how to develop it, we believe that it is present in our curriculum.

		We are exploring aspects of Self Reflection; here are two examples:
		\begin{enumerate}
			\item Development of strong study skills will improving students' ability to self-reflect; consequently, the SAC has discussed adding study skills as formal components of MTH 20 (see \cpageref{cur:sub:mth20}).
			      See \cpageref{cur:sub:studyskills} for more information on study skills material that incorporates student self-reflection.
			\item In AY 2011/12 the Math Learning Assessment Subcommittee (Math LAS) explored Self-Regulated Learning where students were guided in a self-reflective process that helped them evaluate their depth of understanding (or lack thereof).
			      Although Self-Regulated Learning was not incorporated into the assessment activity (due to the complexity and the lack of time remaining), members of the Math LAS who explored this believe it is worthy of future consideration.
		\end{enumerate}
\end{description}

\subsection[Core Outcomes Mapping Matrix]{Update the Core Outcomes Mapping Matrix for your SAC as
	appropriate.\footnote{\url{http://www.pcc.edu/resources/academic/core-outcomes/mapping-index.html}}}
At this time, the Learning Assessment Council expects each LDC SAC to address and assess at least four of the college's Core Outcomes somewhere in their curriculum.
As shown in our current Core Outcomes Mapping Matrix (\vref{sec:app:coreoutcomes}), most of our courses address four Core Outcomes.

To date, most of our outcome assessment activity has been focused in our lower-level courses.
Because of this, it is difficult for us to make truly informed distinctions about variations in outcome levels through our vast array of courses.
We offer many courses at the DE level as well as many courses at the 100 and 200 levels.
While it is obvious that the skills assessed at these various levels are quite different in scope, it is not as clear that the levels of Core Outcome achievement vary in a similar way.

Consider level 4, which is described as ``Demonstrates thorough, effective and/or sophisticated application of knowledge and skills.
''  Does that phrase come with an implied ``appropriate
to the course level being assessed''?
The assignment of levels is very dependent upon the answer to this question.
There are other questions that can be asked that could also greatly impact the way in which levels are assigned.
To wit:
\begin{itemize}
	\item Pitched vs.
	      \ Attained: Should levels be assigned based on ``the level to
	      which the course is taught'' or ``the expected level of attainment for a
	      student who passes with a C (or B or A)''?

	\item Coverage vs.
	      \ Depth: If level 3 or 4 is assigned, should the Core Outcome
	      be a recurring theme in the course or can level 3 or 4 be reached if there is
	      only one or two assignments that have ``depth''?

	\item Prerequisite expectation: If a course has an expected prerequisite knowledge/skill for a particular Core Outcome, which is critical for success in the course, but not covered/developed in the course, what level should be assigned: 1 or 4?

	\item Assessment: Is there an expectation for assessment for all courses based on the assigned level (e.g., perhaps SAC-level assessment is required for courses where a level 3 or 4 is assigned)?

	\item Benchmark comparison: If we rate a course with a 200 designation as ``level 4'' for a particular outcome, is it appropriate to have a developmental education course also at level 4 or should it be at most level 2?

	\item Course-to-course comparison vs.
	      \ stand-alone analysis: Generally, should
	      a course-to-course comparison even be done?
	      Perhaps given the various ways Core Outcomes might manifest, a course-to-course comparison is meaningless (i.e., just as Critical Thinking and Problem Solving could be level 4 for very different reasons across disciplines, it may be possible to have level 4 for very different reasons within a single discipline).
\end{itemize}

\recommendation[LAC, Degrees and Certificates Committee]{We recommend that the
	LAC compose a concise but thorough document that precisely specifies the
	context in which levels should be determined when a SAC constructs its Core
	Outcomes Mapping Matrix.}

\section[Assessment of College Core Outcomes]{For Lower Division Collegiate
  (Transfer) and Developmental Education Disciplines:  Assessment of College Core
  Outcomes    (note:  Please include the full text of your annual reports as
  appendices, and summarize them here).}\label{ass:sec:coreoutcomes}

Our full annual Learning Assessment reports can be found at these links, and are summarized in the pages that follow.
\begin{description}
	\item[2009/10:] \cite{annualLASreport2009}
	\item[2010/11:] \cite{annualLASreport2010}
	\item[2011/12:] \cite{annualLASreport2011}
	\item[2012/13:] \cite{annualLASreport2012}
\end{description}

\subsection[Core Outcome assessment design and process]{Describe the assessment design and processes that are used to
	determine how well students are meeting the College Core Outcomes.}
We will discuss the evolution of the assessments by academic year, starting from 2009/10.
\begin{description}
	\item [2009/10: Critical Thinking  \& Problem Solving]

	      This was the first year the LAC asked SACs to assess a Core Outcome.
	      Our assessment activity was developed by a small group of interested math faculty with minimal coordination with the full SAC.
	      Student artifacts were collected from the interested faculty who volunteered to give the assessment activity in their courses, and so it was not a statistically sound sample.
	      The activity involved both direct and indirect assessment:
	      \begin{itemize}
		      \item The direct assessment involved finding and correcting conceptual, arithmetic, and formatting mistakes in expected procedural skills for MTH 65 and MTH 95.
		            Students must know how to do the problem before they are able to to identify and correct mistakes.
		            This level of analysis is typically very difficult for students and is a high on Bloom's taxonomy.
		      \item The indirect assessment involved asking students to respond to questions like, ``Do you feel this class has improved your critical thinking and problem solving skills?
		            ''
	      \end{itemize}

	\item[2010/11:  Critical Thinking \& Problem Solving and Communication] 

		At the beginning of Fall 2010, the Math SAC decided to create a standing committee to ensure assessment work was a high priority.
		The Math Learning Assessment Subcommittee (Math LAS) was born.
		There were 14 members, most of whom were full-time faculty.
		This was a big improvement from last year's work where only a small group of faculty participated.

		Although our previous year's CT \& PS activity had been high on Bloom's Taxonomy, we decided that procedural skills did not fully capture how we want our student to think critically about mathematics.
		Instead of continuing with the previous year's work, we decided to develop a new activity for CT \& PS.

		Still very new to assessment, we did not know what type of activity might generate the most useful information.
		To help us decide, we developed three assessment activities and collected student artifacts for each activity.
		The chosen activity was randomly given to 12 of the 72 sections of MTH 65 held in Winter 2011.
		(Note: We also created an activity for MTH 244, but there was an error in one of the questions.
		Although a portion of the artifacts were evaluated, we ultimately decided to abandon this attempt and focus our limited resources toward the MTH 65 analysis.)

		For the MTH 65 assessment, we collected 240 student artifacts.
		To help ensure that data would be a true SAC-level assessment (vs.
		\ an evaluation of individual
		instructors), faculty members were instructed to remove identifying information
		from student work and submit their artifacts to an administrative assistant who
		tracked submission of work only.
		Sixteen members of the SAC were normed to the rubric that had been developed by the Math LAS.
		Two members of the LAC's Program Assessment for Learning (PAL) facilitated this work and guided faculty through a trend analysis.
		(Note: Our process and involvement of Math SAC members was so impressive as compared to other large SACs that the Learning Assessment Council awarded the Math SAC an ``Oscar'' at their Spring circus event.)

	\item[2011/12: Self Reflection and Professional Competence] 

		This year we sent a survey to all students enrolled in a math class in the first week of Spring 2012.
		The Math LAS was awarded a LAC grant and we used these funds to hire a consultant, Una Chi, to help us refine the survey and evaluate the student responses to the survey.
		We also discussed the wording of the questions with the DE Reading and Writing faculty members to help ensure the questions would be understood by all students.
		Approximately 2300 students responded to the survey, and the response rates for particular courses mirrored student enrollment in those courses and other demographic information.
		The survey was an indirect measure of students' perceptions.

		For Self Reflection, we focused on questions that we felt would fit the following three areas:
		\begin{enumerate}
			\item Reflection---Core reflective thinking items; autonomy and relatedness aspects from self-determination theory
			\item Orientation---Mastery/performance, internal/external locus of control (hold self responsible vs.
			      holding others responsible)
			\item Competency---Belief about self-ability to perform in math
		\end{enumerate}
		Sample Self Reflection items on the survey:
		\begin{quote}
			I know when I need help on a math concept.
		\end{quote}
		\begin{quote}
			When I get a math test back, my grade is what I expect it to be.
		\end{quote}
		\begin{quote}
			My feelings about math affect my learning of math.
		\end{quote}

		For Professional Competence, we used the suggestions of our LAC coach to craft questions about students' perceptions of math in terms of their future job and career goals.

		Sample Professional Competence items on the survey:
		\begin{quote}
			The skills I learn in a math class are not important to me or my future goals---I just need to pass the course.
		\end{quote}
		\begin{quote}
			In PCC math classes, what knowledge, skills, habits or ways of thinking have you practiced that might help you in the work place?
			[Choices included punctuality,
			problem solving, working in groups, self discipline, career specific math
			skills, interpret graphs/charts]
		\end{quote}
		\begin{quote}
			My career interest requires some mathematical knowledge.
		\end{quote}

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence] % On page 13 and 14 you discuss how individual course outcomes are not
		% addressed due to the LAC focus on college core outcomes.  But this year
		% y'all totally addressed individual course outcomes, in the context of the
		% CT & PS core outcome.  Plus it was doubly recognized with awards.  I think
		% you should pull out this years' work as specifically covering course
		% outcomes right upfront on page 13.

		This was our third investigation into Critical Thinking \& Problem Solving.
		Our previous attempt had given us rich data, but this year we decided to investigate students' ability to solve nine math problems that specifically represent the topics covered in MTH 95 that we consider essential for success in MTH 111.
		The assessment activity was administered to every face-to-face section of MTH 95 at all PCC campuses in the Winter of 2013.
		We collected 677 student responses from 33 different sections across the college; all identifying information for both instructors and students were removed.

		The math problems included in the assessment were selected by faculty from our current MTH 95 textbook:
		\begin{itemize}
			\item For Critical Thinking \& Problem Solving: We incorporated problems that contain units and involve a real-world context.
			\item For Professional Competence:  We incorporated problems that emphasize the content needed to be successful in the next course, MTH 111.
			      For this activity, Professional Competence was interpreted as the, ``knowledge, skills and attitudes necessary to enter and succeed in a defined profession or advanced academic program'' (\cite{coreoutcomes}).
		\end{itemize}
		During the LAC's summer peer review process, our report won awards in two categories: ``Assessment Design'' and ``Planned Improvements to Increase Student Attainment of Outcomes''.
		The awards were announced at 2013 SAC Chair Inservice.
		\footnote{\awardsurl}
\end{description}

\subsection[Core Outcome assessment results]{Summarize the results of assessments of the Core Outcomes.}
As a reminder, the full reports for our assessment activity are available using the links on \cpageref{ass:sec:coreoutcomes}.

\begin{description}
	\item[2009/10: Critical Thinking \& Problem Solving] We did not evaluate the student artifacts due to lack of time during the 2009/10
		academic year.
		We intended on completing the work during 2010/11; however, since we did not have a statistically valid sample and since we wished to try a different type of assessment during 2010/11, we decided not to evaluate these artifacts.
		Even though we did not evaluate student learning, the faculty members gleaned valuable information about the process of assessment.
		The assessment artifacts have been saved in case the Math LAS wishes to review them to guide future work.
	\item[2010/11:  Critical Thinking \& Problem Solving and Communication] \Cref{ass:tab:201011scores} presents the results of the rubric scores for our
		2011 assessment of Critical Thinking \& Problem Solving and Communication in 13
		sections of MTH 65.
		\begin{table}[!htb]
			\centering
			\caption{2010/11 Assessment Scores}\label{ass:tab:201011scores}
			\begin{tabularx}{\linewidth}{Xlll}
				\toprule
				Rubric Score  & 1 or 2               & 3                 & 4 or 5                 \\
				              & (below expectations) & (met expectation) & (exceeded expectation) \\
				\midrule
				CT \& PS      & 55\%                 & 35\%              & 10\%                   \\
				Communication & 50\%                 & 28\%              & 22\%                   \\
				\bottomrule
			\end{tabularx}
		\end{table}

		We realized that the rubric scores did not tell us specific information (e.g., \emph{why} did the artifact score ``below expectation'' or ``exceeded expectation''?
		).
		The LAC Program Assessment for Learning  facilitators suggested that we do a trend analysis which produced more meaningful information.
		Here are some results of the trend analysis:
		\begin{itemize}
			\item Many students did not seem to realize that not all data are linear.
			\item Many students were not able to give a well-supported conclusion.
			\item Students typically do not represent equivalence correctly.
			\item Many students incorrectly applied the idea of percentage.
		\end{itemize}

		%%Alex is here

	\item[2011/12: Self Reflection and Professional Competence] For Self Reflection:
		\begin{itemize}
			\item Students are not self-critical enough.
			\item There was a significant group mean difference between self-reported grade and self-reflection behavior on all grade level differences.
			      Note: ``grade level'' is the self-reported grade (A, B, C, D, F, P, NP, other) for the student's previous math class.
			\item There is a clear difference in the reflective thinking ability of students in high-level math courses versus students in low-level math courses (like MTH 20).
		\end{itemize}
		For Professional Competence:
		\begin{itemize}
			\item It was surprising that Engineering was chosen as ``my career interest'' by 11\% of respondents (the plurality).
			      Nursing and Business were both in second at 7\%.
			\item Presentation skills ranked low by students as helpful in the workplace.
			      A good SAC discussion could center around this; we may not wish to ``force'' students to perform presentations in lower level courses where math anxiety is increased.
		\end{itemize}

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence] % FWIW, I think you should offer this evaluation again (though of course
		% modified based on what you've learned) if only to answer your own questions
		% about why they did so horribly.

		Our assessment consisted of nine math problems.
		After all submissions were graded, the average score was approximately 3.8 out of 9.
		On a class-to-class basis the average had a low of 1.7 out of 9 to a high of 5.4 out of 9.
		We were unsure of why the average of all classes was so low, and we found it alarming.
		Did students not take the activity seriously since most instructors did not assign points to it?
		Would it have been better to give it with the final exam when most students were prepared for a mathematics assessment?

		Data in more detail is presented graphically and with tables in section 3 of the full report, \cite{annualLASreport2012}.
		Some summary points follow:
		\begin{itemize}
			\item The problems that involve working with function notation were answered correctly at a much lower rate than we expected given the amount of time dedicated to that topic in MTH 95.
			\item A problem that involved linear equations was also answered correctly at a much lower rate than we expected, as that topic is covered in MTH 60, 65 and 95.
			\item Given that time constraints made it difficult to discern a student's conceptual understanding of a topic and our decision to mark answers as correct or incorrect (with no ``partial credit''), we should consider altering this activity if used again.
		\end{itemize}
\end{description}

\subsection[Assessment-driven changes]{Identify and give examples of assessment-driven changes that have
	been made to improve students' attainment of the Core Outcomes.}

Below we discuss the assessment-driven changes that came out of our annual assessment projects in academic years 2009/10, 2010/11, 2011/12, and 2012/13: 

\begin{description}
	\item[2009/10: Critical Thinking \& Problem Solving] 

		We chose to start anew rather than analyze data that was not statistically significant.
		No course-level assessment driven changes came from this years work, though the SAC did significantly modify its approach to assessment in following years due to this first year finding.

	\item[2010/11:  Critical Thinking \& Problem Solving and Communication] 

		During 2010/11, it became clear that the Math LAS members would not be able to develop/conduct assessment \emph{and} lead the SAC with assessment-driven changes from the prior year's work.
		At the end of this academic year, the SAC created another assessment standing committee, called the Action Subcommittee.
		This subcommittee will take the results and the recommendations from the previous year's Math LAS research and lead the SAC in deciding what should be changed and how to implement that change.
		For 2010/11 work, only individual changes to instruction occurred from the assessment results.
		(See section 1 from the full report, \cite{annualLASreport2010}.)

	\item[2011/12: Self Reflection and Professional Competence] 

		After brainstorming a list of actionable items from the 2012 research, the Action Subcommittee decided to work on the following item: ``disseminate successful ideas already used by our faculty for improving self-reflection via study skills and student-centered learning.
		''  The goal was to create activities that faculty
		could easily incorporate into their classes that would help students develop
		self-reflection behaviors that would lead to better study skills.
		Math SAC members were asked to submit activities that were already being used successfully, and the committee received 25 different activities.
		During an all-day SAC meeting we split into breakout sessions and each group was asked to look over the activities and create a list of best practices for incorporating them into the classroom.
		These worksheets are available for instructors to download and incorporate into their classes at \cite{selfcenteredlearning}.
		Additionally, a faculty member created a series of self-reflection and study skills videos that are being used in a lot of developmental classes (see \cpageref{cur:sub:studyskills}).

	\item[2012/13:  Critical Thinking \& Problem Solving and Professional Competence] 

		For the complete list of actionable items from this year's research, see section 4 in the full report, \cite{annualLASreport2012}.
		\begin{itemize}
			\item Add to the CCOGs the expectation that students check the reasonableness of their results (e.g., a result of $-5$ or $1{,}000{,}496$ would not be a reasonable result for ``the number of hours driven on a weekend trip'').
			      Ideally, students should be encouraged to develop a habit of verifying their results regardless of whether or not the problem is a contextualized problem or not.
			\item Create a minimum skills test for MTH 95.
			\item Discuss methods of course content delivery in a way that supports both full- and part-time faculty.
			\item Form a Developmental Math Committee that will research different ways we might be able to redesign our pre-college curriculum in order to better prepare students for college level math as well as better serve students in CTE programs.
		\end{itemize}

\end{description}

The Action Subcommittee is currently reviewing the 2012/13 assessment work and may propose to the SAC other ideas for implementation.

